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BACKGROUND 


Trait  rating*  by  paers  are  frequently  used  to  assess  the  relative  status  of  the  individ¬ 
ual  being  rated  on  a  given  trait.  The  use  of  peer  ratings  in  practical  applications  in 
the  military  owes  much  to  Jacob  L.  Moreno,  who  used  the  term  "sociometric  measurement"  to 
apply  to  such  ratings.  A  sociometric  measurement  is  defined  as  any  attempt  to  measure  a 
functioning  social  group  or  organizational  unit. 

Scott  (1965)  distinguished  several  components  which  are  present,  potentially,  in  the  so¬ 
ciometric  measurement  of  any  group.  He  had  members  of  10  fraternities  and  sororities  of 
the  University  of  Colorado  rate  their  fellow  fraternity  (or  sorority)  members  on  four 
traits i  Academic  orientation  (how  seriously  they  take  their  school  work))  organization 
orientation  (their  contribution  to  house  activities) i  social  orientation  (their  ability 
to  get  along  with  other  kinds  of  people) >  and  friendship  (how  much  you  like  them) .  For 
each  organization  and  each  trait  separately,  a  sociomatrix  was  constructed,  summarizing 
the  rating  given  by  each  person  in  the  organization  to  every  other  person  (pp.  120-121) . 

Figure  1  illustrates  a  typical  sociomatrix  for  a  small  group  of  individuals.  Since  the 
analysis  is  standard,  it  does  not  matter  which  particular  trait  was  being  rated. 
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*A  person  does  not  rate  himself. 
FIGURE  1.  HYPOTHETICAL  SOCIOMATRIX 


In  Scott's  analysis,  three  basic  components  can  be  distinguished  in  any  Sociomatrix, 
which  he  termed  status,  bias,  and  mutuality.  Status  is  the  component  which  reflects  in¬ 
dividual  difference  in  perceived  ability  on  a  trait.  In  figure  1,  individual  ratees 
differ  in  their  relative  standing  on  the  trait  being  rated.  (Column  sums  of  27,  25,  etc., 
reflect  status  differences  on  the  trait.)  Bias  is  the  component  which  reflects  differ¬ 
ences  between  raters  in  the  way  they  use  the  rating  scale.  In  Figure  1,  rater  number  5 
tended  to  rate  everyone  low,  whereas  rater  number  02  tended  to  rate  his  peer  high.  Indi¬ 
vidual  differences  in  row  sums  reflect  rater  bias.  Mutuality  is  the  component  which 
reflects  degree  of  similarity  between  rater  and  ratee.  For  example,  in  Figure  1  rater  01 
gives  a  rating  of  10  to  rater  02,  who  in  turn  gives  a  rating  of  10  to  rater  01.  Accord¬ 
ing  to  Scott,  these  three  components  (peer  statUB,  rater  bias,  and  mutuality)  are  poten¬ 
tially  present  in  any  sociometric  trait  rating,  and  they  are  not  independent. 

Scott  developed  statistical  indices  of  the  amount  of  variance  in  any  sociomatrix  which  is 
associated  with  each  of  these  components. 

Nest  Point's  leadership  evaluation  system  routinely  collects  sociometric  data.  Men  rank 
order  the  top  one-fourth  and  the  bottom  one-fourth  of  the  others  in  their  company  and 
class,  using  USMA  Form  2-320.  These  rank  orders  can  be  converted  into  sociometric  ratings 
of  the  type  studied  by  Scott.  Cadets  are  required  to  rate  their  peers  on  "ability  to 
command,”  and  supposedly  give  their  own  independent,  impartial,  and  objective  judgment. 

It  is  not  known  to  what  extent  such  judgments  are  influenced  by  personal  friendship 


among  cadets.  Whether  by  an  unconscious  inflating  of  the  estimated  abilities  of  one's 
friends,  or  by  a  direct  collusion  ("I'll  rate  you  high  and  you  rate  me  high"),  a  mutuality 
bias  is  always  potentially  available  to  cadets  in  making  their  ratings.  This  study  was 
designed  to  investigate  the  extent  of  mutuality  bias  in  LES  ratings  of  the  first  rating 
period,  AY  74-75,  for  the  Class  of  1976  peer  ratings. 


METHOD 

Patai  Type  2  ratings  made  by  the  Class  of  1976  on  the  Class  of  1976  were  collected  in 
November  1974,  on  USMA  Form  2-320.  There  were  886  raters  and  888  ratees. 

Analysis :  In  addition  to  the  usual  procedure,  cadets  were  instructed  to  record  their  own 
man  number  at  the  bottom  of  the  rating  sheet.  These  numbers  were  keypunched  by  AMIC;  and 
after  processing  the  MP311  grades,  the  punched  cards  were  released  to  the  Director  of 
Institutional  Research.  A  computer  program  was  then  written  to  convert  the  cards  into  a 
sociomatrix.  The  program  assigns  the  number  "1"  to  the  man  who  is  selected  as  "best" 
able  to  command)  successive  nominations  are  given  higher  rank  numbers.  In  a  group  of  N 
persons,  those  who  are  not  nominated  by  a  rater  are  assigned  an  average  rank  of  (N+l)/2. 

A  second  program  computes  measures  of  peer  status,  rater  bias,  and  mutuality.  ;Scott, 
1965,  pp.  363-376,  gives  the  formulas  for  these  terms) .  Each  of  these  three  measures  is 
an  intra-class  correlation  (eta  squared) ,  expressing  the  proportion  of  the  variance  in 
the  sociomatrix  which  can  be  attributed  to  each  component.  As  applied  to  the  USMA  data, 
Scott’s  "peer  status"  measure  is  an  index  of  leadership  dif ferentation  within  the  com¬ 
pany;  for  this  reason,  it  is  called  the  leadership  index  in  the  remainder  of  this  report. 
As  applied  to  the  USMA  data,  there  is  no  rater  bias,  since  all  rows  add  to  a  constant. 
Thus,  the  rater  bias  measure  is  virtually  zero  and  is  not  reported.  Scott  distinguishes 
among  three  of  mutuality:  total  mutuality,  status-bias  concordance,  and  differential 
mutuality.  Total  mutuality  reflects  the  similarity  in  ratings  between  all  pairs  of 
raters  and  ratees.  Total  mutuality  is  high  if  the  between  pairs  variance  is  high  rela¬ 
tive  to  the  within-rair  rating  variance.  Within-pairs  rating  variance  depends  on  how 
similarly  the  two  men  in  the  pair  rate  each  other.  Status-bias  concordance  is  the  tend¬ 
ency  of  raters  with  high  status  to  have  high  rater  bias.  In  the  USMA  LES  data,  status- 
bias  concordance  was  zero,  due  to  the  method  of  collecting  LES  ratings,  as  noted  above 
for  rater  bias.  Differential  mutuality,  like  status-bias  concordance,  is  a  subcomponent 
of  total  mutuality.  It  represents  the  degree  to  which  a  person  departs  from  his  general 
bias  in  rating  a  colleague  in  the  same  manner  that  that  colleague’s  rating  of  the  person 
departs  from  the  person's  general  leadership. 
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The  program  was  verified  on  the  sample  data  given  by  Scott  on  p.  276,  so  it  is  accurate. 

To  aid  the  interpretation  of  the  results  of  the  analysis,  two  artificial  sociometric  ma¬ 
trices  were  constructed,  each  composed  of  the  responses  of  20  hypothetical  cadets.  Matrix 
One  (See  Appendix  A)  was  constructed  with  the  assumption  that  all  cadets  (1)  knew  who  the 
best  leaders  were  and  (2)  agreed  perfectly.  In  Matrix  One,  the  leadership  index  was  .98, 
very  high,  as  expected.  The  differential  mutuality  index  was  .35,  and  the  total  mutual¬ 
ity  was  .05.  Note  that  these  indices  add  to  more  than  100%  of  the  variance,  because  they 
are  not  mutually  orthogonal.  Also,  it  should  be  noted  that  some  differential  mutuality 
is  to  be  expected,  since  leaders  will  rate  other  leaders  high  on  an  objective  basis,  even 
if  there  is  no  personal  friendship.  The  total  mutuality  index  corrects  for  mutuality 
among  leaders  and  is  substantially  lower  than  the  differential  mutuality  index.  Because 
the  total  mutuality  index  appears  to  suppress  overlapping  variance  more  completely,  it  is 
the  one  mutuality  index  used  in  the  remainder  of  this  report.  Matrix  Two  was  constructed 
under  the  assumption  that  (1)  every  cadet  nominated  his  best  friends  as  having  top  leader¬ 
ship  (2)  cadets  pair  off  in  such  a  way  as  to  virtually  eliminate  status  differences.  The 
pattern  of  choices  in  this  matrix  is  known  as  a  circumplex,  since  man  numbers  can  be  ar¬ 
ranged  in  a  circle  with  highest  ranking  choices  being  given  to  the  nearest^neighbors  in 
the  circle  (see  Appendix  B) .  In  Matrix  Two,  the  leadership  index  was  -.051  and  mutuality 
was  .93.  Both  numbers  are  in  the  correct  order  of  magnitude  predicted  for  Matrix  Two. 
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CONSENSUS  ON  LEADERSHIP  AND  MUTUALITY  IN  PEER  RATINGS 
CLASS  OF  1976,  NOV  74  RATINGS 


COMPANY 

N 

PROPORTION  OF  VARIANCE 
LEADERSHIP 

IN  RATINGS, 
MUTUALITY 

A2 

21 

.55 

.08 

E4 

23 

.54 

.16 

D2 

26 

.52 

.12 

F3 

22 

.50 

.11 

11 

23 

.49 

.14 

C3 

31 

.46 

.14 

D1 

19 

.45 

.13 

HI 

21 

.44 

.09 

B2h 

19 

.42 

.17 

E2b 

28 

.39 

.10 

12 

27 

.38 

.11 

E3 

30 

.38 

.19 

Cl 

26 

.38 

.16 

13 

23 

.38 

.19 

G1 

22 

.36 

.07 

B4 

27 

.33 

.18 

G3 

27 

.32 

.15 

El 

29 

.32 

.23 

H4 

27 

.31 

.17 

G4 

26 

.30 

.16 

F4 

23 

.30 

.19 

H3 

25 

.29 

.16 

B1 

24 

.29 

.23 

B3 

21 

.27 

.11 

A3 

27 

.27 

.22 

D4 

22 

.27 

.26 

C4 

29 

.27 

.14 

H2 

32 

.27 

.12 

FI 

23 

.26 

.21 

G2 

24 

.25 

.29* 

C2 

22 

.25 

.11 

14 

20 

.24 

.25* 

A4 

25 

.24 

.30* 

F2 

21 

.20 

.12 

D3 

29 

.20 

.23* 

A1 

24 

.17 

.33* 

MEDIAN 

.315 

.160 

'Indicates  Mutuality  is  greater  than  Leadership. 

aBoth  Leadership  and  Mutuality  indices  are  intra-class  correlations  defined  by 
Scott  (1969,  p274) . 

bIrregularity  in  the  data:  Not  all  cadets  nominated  the  required  number  in  their 
company . 


RESULTS 


Table  1  shows  the  leadership  and  mutuality  index  for  each  of  the  36  companies  in  the 
Class  of  1976,  first  detail.  Both  indices  are  to  be  interpreted  as  intraclass  correla¬ 
tions  (eta  squared) ,  and  hence  indicate  the  proportion  of  variance  in  the  ratings  due  to 
each  source  of  variance.  In  the  case  of  leadership,  the  index  shows  the  percentage  of 
variance  in  aociometric  ranks  which  are  associated  with  individual  differences  in  peer- 
perceived  leadership  ability.  It  represents  the  degree  to  which  a  group  of  cadets  agree 
among  themselves  in  rank  ordering  informal  leaders.  In  mutuality,  the  index  represents 
the  degree  to  which  pairs  of  cadets  tend  to  assign  similar  leadership  ranks  to  each  other. 

Table  1  shows  that  in  thirty-one  costpanies  the  leadership  index  was  higher  than  the  mu¬ 
tuality  index.  In  five  companies,  the  mutuality  index  was  higher.  These  results  suggest 
that  most  groups  of  cadets  do  not  let  mutuality  influence  their  ratings  to  any  great  ex¬ 
tent. 

The  median  proportion  of  variance  due  to  consensus  on  leadership  is  .315.  On  the  average, 
the  concensus  is  moderate  at  best,  and  there  is  room  for  a  large  increase  in  consensus  on 
leadership  with  each  company. 

There  appears  to  be  a  considerable  degree  of  variation  among  companies  in  the  leadership 
index:  for  Company  A2,  it  is  .55,  but  for  Company  Al,  it  is  only  .17.  If  we  can  assume 
that  the  sampling  distribution  of  the  index  is  normal,  then  the  difference  between  Com¬ 
panies  Al  and  A2  may  be  tested  with  the  ordinary  F  ratio.  If  so,  the  difference  is  sig¬ 
nificant  at  the  .01  level  (F»3.23,  DF"21/24) .  Of  course,  there  is  no  way  to  establish 
conclusively  that  companies  differ  in  leadership  consensus  without  further  research.  If 
the  leadership  index  is  stable  from  the  first  period  rating  to  the  second  period  rating, 
then  we  might  be  in  a  better  position  to  accept  the  differences  between  companies  as  real. 
Or,  if  the  leadership  index  correlates  with  company  morale,  with  cadet  intramural  perform¬ 
ance,  or  with  other  company  level  indicators,  then  the  differences  among  companies  might 
be  regarded  as  genuine.  Further  research  would  help  to  verify  the  results  of  the  present 
study . 

Table  1  shows  that  mutuality  and  leadership  are  inversely  related.  The  correlation  is 
-.49.  As  one  increases,  the  other  tends  to  decrease.  Such  results  are  consistent  with 
theory,  since  mutuality  should  interfere  with  leadership  differentiation  within  the  com- 
pany. 

Table  2  shows  the  magnitude  of  leadership  index  for  five  different  traits.  The  first 
trait,  "ability  to  command,"  summarizes  the  leadership  index  data  for  USMA  cadets.  The 
other  four  traits  were  studies  by  Scott  (1965,  p.  124)  at  Colorado,  and  were  described 
above.  A  chi-square  test  was  performed  to  contrast  the  five  traits  and  was  significant. 
This  indicates  that  consensus  within  a  group  in  ranking  peer  status  on  different  traits 
depends  on  the  type  of  trait  being  rated.  For  example,  the  peer  status  index  is  lower 
for  "friendship"  as  a  sociometric  criterion  than  for  "contribution  to  house  activities." 
Thus,  group  members  tend  to  achieve  a  higher  consensus  when  they  rate  the  latter  trait 
than  when  they  rate  friendship.  It  is  worth  noting  that  the  USMA  criterion,  "ability  to 
cosmand  a  group  of  men,"  generally  elicits  intermediate  degrees  of  consensus  on  the 
leadership  index.  There  is  significantly  more  consensus  on  "ability  to  command"  than  on 
"friendship"  (X2  "  4.2,  ldf,  p<.05). 
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TABLE  2 

LEADERSHIP  INDEX  FOR  FIVE  TRAIiS 


Magnitude  of  Leadership  Index 


Trait 

.30  or 

Below 

.  31  or 
Above 

Number 
of  Groups 

1. 

Ability  to  command  a  group  of  mena 

17 

19 

36 

2. 

How  much  you  like  them  (Friendship)*3 

9 

1 

10 

3. 

How  seriously  they  take  their  schoolwork*3 

3 

7 

10 

4. 

Their  contribution  to  house  activities*3 

2 

8 

10 

5. 

Their  ability  to  get  along  with  other  people*3 

8 

2 

10 

X2  -  15.3,  4df,  p<.01. 
a Source:  Table  1,  this  report. 
^Source;  Scott  (1965),  p.  124. 


The  analysis  of  Table  2  supports  the  hypothesis  that  the  USMA  leadership  ratings,  which 
should  reflect  "ability  to  command,"  are  not  strongly  biased  by  friendship  patterns  among 
cadets. 

DISCUSSION 

The  degree  of  consensus  on  leadership  ranking  is  usually  discussed  under  the  general 
heading  of  reliability.  The  concept  of  reliability  has  several  different  meanings,  in¬ 
cluding  consistency  over  time  as  well  as  inter-judge  consistency.  Inter- judge  consistency 
should  be  considered  as  a  prerequisite  to  all  other  forms  of  sociometric  reliability 
(Lindcey  and  Byrne,  1968) .  Unfortunately,  inter-judge  consistency  has  been  ignored  in 
prior  research  at  the  Military  Academy.  For  example,  Tobin  and  Marcrum  (1967)  discuss 
only  stability  of  total  ASR  scores  over  time.  It  has  been  demonstrated  many  times  over 
that  the  LES  is  highly  stable  over  time.  In  a  recent  study  of  the  Class  of  1977,  the 
November  peer  rating  component  of  LES  had  a  correlation  of  .70  with  the  March  rating 
(Priest,  1975,  p.  6).  Thus,  the  high  stability  of  peer  rating  has  led  to  the  neglect  of 
inter- judge  consistency.  Outside  the  Military  Academy,  correlations  between  measures  of 
leadership  status  for  random  halves  of  a  group  are  "often  in  the  .80's  and  .90' s" 

(Lindsey  and  Byrne,  1968) .  Considering  that  the  average  intra-class  correlation  in 
Table  1  was  .315  (and  the  square  root  of  this  is  .56),  it  would  seem  that  the  inter-rater 
consistency  of  cadets  within  companies  is  low. 

The  present  study  provides  additional  objective  evidence  on  the  relationship  between  peer 
leadership  ratings  and  friendships.  An  earlier  reviewer  had  asserted,  on  the  basis  of 
incomplete  evidence,  that  "peer  nominations  are  not  adversely  affected  by  considerations 
of  friendship"  (Nadal,  1968,  p.  7) .  It  is  no  longer  necessary  to  extrapolate  from  re¬ 
search  done  at  other  institutions.  The  evidence  in  this  study  is,  of  course,  indirect, 
in  that  we  did  not  ask  cadets  about  their  friendship  choices;  nevertheless,  if  mutual 
friendship  were  a  strong  factor  in  leadership  nominations,  it  would  have  interfered  more 
with  consensus  in  leadership.  Since  the  later  interference  was  not  observed,  we  can 
conclude  that  friendship  is  not  a  strong  factor  in  peer  nominations  at  USMA. 

The  use  of  a  variety  of  traits  for  rating  leadership  has  not  been  systematically  explored 
at  the  Military  Academy.  Lindcey  and  Byrne  (1968)  describe  several  studies  in  a  miscel¬ 
laneous  variety  of  situations  where  different  sociometric  items  were  "substantially" 
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correlated,  with  r’s  ranging  between  .50  and  .83  (p.  479).  This  suggests  that  the  spe¬ 
cific  wording  "ability  to  command  a  group  of  men"  is  not  essential  for  valid  rating  of 
leadership.  Other  wording  such  as  "ability  to  command,”  or  "contribution  to  corps-squad 
activities”  might  be  tried.  The  full  wording  of  the  definition  of  "leadership"  given  on 
USMA  Form  2-320  (ability  to  command  a  group  of  men  in  the  accomplishment  of  an  assigned 
mission  which  maintaining  within  the  group  high  standards  of  discipline,  morale,  and  per¬ 
sonal  morality)  is  complex.  Perhaps  a  simpler  definition  of  the  criterion  for  rating 
would  lead  to  higher  consensus  on  leadership  status.  That,  in  turn,  would  lead  to  higher 
reliability  of  the  LES  ratings. 


Further  research  is  needed  to  determine  precisely  the  factors  which  influence  peer  con¬ 
sensus  at  the  company  level.  Because  the  major  function  of  peer  ratings  is  to  rank  order 
individuals  in  leadership  ability,  the  effects  of  group  dynamics  on  peer  rating  have  been 
ignored.  A  better  understanding  of  the  group  dynamics  of  peer  rating  is  needed.  It  is 
suggested  that  further  research  be  undertaken  to:  (1)  determine  the  stability  of  the 
leadership  consensus  index  from  the  first  to  second  rating  period;  (2)  determine  the  im¬ 
pact  of  intramural  wins  and  losses  on  leadership  consensus;  and  (3)  determine  the  rela¬ 
tionship  between  the  self-image  or  morale  of  the  company  and  its  leadership  consensus. 

The  present  research  design  could  easily  be  extended  to  cover  the  entire  Corps  of  Cadets 
for  the  1976-77  academic  year,  with  virtually  no  increase  in  cadet  testing  time:  the 
present  design  is  unobtrusive.  Such  research  could  add  to  our  understanding  of  the  group 
dynamics  of  individual  companies,  and  would  be  particularly  valuable  for  analyzing 
changes  in  group  dynamics  that  might  be  introduced  by  changes  in  the  Corps  of  Cadets, 
such  as  the  admission  of  women. 
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REFERENCE  NOTES 


^Except  for  the  small  negative  factor  used  to  correct  sample  statistics  for  parameter 
estimation. 
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*A  copy  of  the  program  is  available  at  the  Office  of  the  Director  of  Institutional 
Research. 
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MATRIX  ONE,  A  HYPOTHETICAL  EXAMPLE  OF  PERFECT  PEER  CONSENSUS  ON  LEADERSHIP 


APPENDIX 


In  the  West  Point  leadership  ranking  system,  which  this  example  simulates,  persons  do  not  rate  themselves 
certain  others;  such  cases  are  designated  by  a  rating  of  "10"  in  this  matrix. 
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